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PROCEDURES FOR EMPIRICAL DETERMINATION OF EN-ROUTE CRITERION LEVELS . 

Michael H, Moncrief 
SWRL Educational Research and Development 

ABSTRACT . 

■ • . > ■ . - ... . ' 

En*route Criterion Levels (ECLs) are defined as decision rules for 
'• ' * ' • ■ . . ■ ^ 

z ■ ■ * ■ 

predicting pupil readiness to advance through an instructional sequence. 

This study investigated the validity of present ELCs in an indivi^^f^ized 

mathematics program and tested procedures for empirically determining 

optimal ECLs. Retest scores and subsequent progress were validating 

criteria. Results indicated empirical data can identify more efficient 

^* • ■ 

ECLs than those established a 'priori ^ To justify the cost involved/ 

~f ■ ' ' -^^ . 

however, such data should be collected by .the instructional designers 

during the formative evaluation field test of a program, using automated 

data processing and multiple-marrix sampling techniques. 
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PROCEDURES FOR EMPIRICAL DETEBM^ATION OF EN-ROUTE CRITERION LEVELS 

f. 

Hlchael H. Monqrief 

Recently there has been Increased discussion among educators about 

designing Instruction for mastery learning. This. discussion has often 

revealed a lack of clarity as to -the meaning of mastery. 

In an educational context mastery of a taskT' generally is considered 

to have been attainted (a) when an individual demonstrates sufficients 

proficiency t6 perform a given^ function, or (b) when he can perform veil 

enough on that task to benefit from being advanced to the next. 

Using the first criterion/ it seems reasonable to say that a 

student has ^'mastered" a given instructional objective when he can be 

expected^to perform satisfactorily in those situations typically found 

in his everyday life that call for the use of t^at which has been 

learned. This, means he m^ist be able to retain and apply some minimal 

amount of what he has^^arned. 

The other educational criterion used In defining mastery is the 

>ne moat frequently used and is much easier t^ determine empirically. 

|As.Glaser (1963) suggests., "m&st^ry" can be'^used to specify the minimum 

''proficiency the^ student ni^eds to demonstrate before 'going on tp the 
■ . j • ' . ■ ■ ■ ■' 

i' next instructional un£t in a sequence. Using thfs definition the most 

^! efficient operationalizatlon of mastery would seem to be that proficiency 
j level which maximizes the subsequent progress of .students jthroyigh the 
I Instructional units. v V ^ * 

I. Currently available individualized instructional programs have 

i •pecifled their mastery levels,' and thus their Instructional; management 



\. 



decision c^rlteria, mainly on the basis of Intultiv^ judgment. Some 

■■' • * * i 

empirical data are needed for use In establishing ymastery levels" 
vhlch maximize the ef^lclenc)s^of selected Instztact^onal management 
decisions. 

V ,Thls study investigated the validity, as Inst^ctlonal management 
decision rules » of the preset en - route criterion 
associated with a selected group of Instruct ipnal 
individualised mathematics program. The vnlldatloji was conducted 
In terms of delayed retest scores and In terms of 

through the Instructional continuum. The emplrlcjil data gathered In 
this study were used to suggest optimal performance 



jlevels (ECL)' 
objectives In an 



subsequent progress 



:e stj^ndards for use 
with the selected objectives. The study also examined the cost/benefit 




oclated with the use of ECLs derived by emplrljcal procedures. 



Individualised 
illrected only at a 
<: .manageicvent tasks of 



Explanation of Terms >^ 
There are many tasks Involved In managing ati 
instructional program.^ This study, however, was 
ver^ specific subset of those tasks. , The specif 



Interest were those concerned with directing studont progress through 



determine whether 



a continuous individualized instructional prograojij In this study 
the term instructional management*V is used to ?4f^^ these 
management tasks*, "Instructional management deelnions" refer to ^ 
those decisions, made on the basis of ECLs, whicl 
or not a student Is ready to advance to the next i^eamlng task in a 
sequence . 

In the context of most individualized instrtle^tional "prbgrdmsf. 
Including the one studied here, instructional marigement decisions are 



m&de on the basis of crtMrlon^refereiaced tests* The8<S( tests are designed 
to measure a student's performance In a prespecl^led domain that has 
been operationally defined* The score on a criterlon-'referenced test Is 
thought to 'Indicate the degree of proficiency that a student has attained 
pn that specific objective. In order to proceed from one learning 
objective to the next^ a student must^ demonstrate a certain degree of 
proficiency on the first objective. I.e., he^ must meet or surpassT the 
perfozmance standards or criterion level for that object lve« Such 
proficiency levels are often referred to as "mastery** levels and a 
student who scores at that level or above Is said to have attained 
mastery* 

The use of mastery levels Is found in many of the recently developed 
Instructional programs. For |nstructlon usltig mastery levels the 
typical paradigm. Illustrated In Figure 1, requires the student to 
att^aln a predetermined criterion score on each objective or unit of 
Instruction before advancing to the next unit f in the sequence* UsuaXly 
0xxy studdtit who falls to reach the criterion is recycled through that 
segment of the program, often receiving some form of remedial instruction. 
Recycling is continued until the desired proficiency Is demonstrated. 
This is the procedure followed by most of the current individualized 
instructional programs. Including Individually Prescribed Instruction 
(Cooley and Glaser, 1969), Project PLAN - Program of Learning in 
Accordance with Needs (Flanagan, 1968) and individualized Mathematics 
System (Ironside, 1971). It is also the paradigm used u most Computer 
Managed Instruction projects (Lawler, 1971) • 



start 

i 



Identify student ' s initial] 
placCTient in the program > | 



Pre-test to diagnose areas of 
weakness in upcoming unit* 
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>re-test indicate"^, yes 
student needs to work 
on unit? 



no 
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attainment of all objectives 
within that unit. • ^ 
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: yes 
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Figure 1. --General Paradigm of the Instructional Management 
Process in Individuialized Instructional Programs 



Ihere are some very real costs involved In procedures requiring 
the use of mastery t^^sts, but there »e even greater costs involved in 
any incorrect instructional decisions umde on the basis of these tests* 
In most individualissed programs, for example, a student who' scoires just ^ 
below the required level is recycled through that^^^iriistnictional unit. 
This recycling could be considered, unnecessary if, had he ^<eten advanced, 
his subsequent progress through the program essentially would be unchanged. 
This procedure incurs unjuf^tif ied costs to the^ system in terms of 
instructional time, the unnecessary use of materials, limiting the i 
progress of the student, and a decrease in his motivational, level. Much 
the same cost also would be involved if a student "wei^ allowed to advance 
bej^ore he had sufficient proficiency in the prerequisite sicllls needed 
for a reasonable probability of success on the subsequent unit, 

' * r' 

To date, mastery levels have been set on an a priori basis. To 
dptlmize the probability of making correct ihstructlonal decisions an 
empirical basis for establishing criterion scores is needed. If empirical 
procedures are found to produce decision points which are significantly 
different from those established on an intuitive or. purely "theoretical" 
basis then the" application of those procedures coul<d make ^ substantial* 
improvement in the operation of individualized instructional programs.. 
Such procedures applied to appropriate instructional programs could be 
expected to facilitate the progress of students through those instructional 
sequences and thus reduce instructional costs. Whether Qr not^hia 
reduction represents a substantial savings largely depends on ^he costs 
of applying the empirical procedures. r The current practice in many 



Instructional programs is to establish the same criterion levels for 
all students- over' all objectives^ Some evidence is needed pertaining 
to the soundness of requiring uniform levels of proficiency over all 

, " - ; •■ ■■■ ■ ■ ^ ' o - 

objectives and over all students. As the number of individualized ^ 
programs of instruction being developed and implemented increases /3o 

does the need to come to grips with these instructional .design questions. 

' ■ , ■ •'. ft • ■ 

Criterion* Referenced Tests and Instructional Management 

The question of the appropriate uses of criterion* referenced and of 

., ., - ., , . , . . .. . , 

norm«j:0xerenced measures is stil^l somewhat unsettled. Convincing arguments 
however > have been presented for the use of csfiterion* referenced assess*- 
nent procedures in making instructional management decisions (6iaser» 
1963; Gagne^ 1965; Fopham and Husek, 1959) « Criterion-referenced tests 
are designed to provide information about the degree of competency attained 
by individual students irr^spectiver of the performance of ojtihersv The , 
use of su^h tests is required in programs where "mastery learning" is r 
the goal /(Airasian, in individualised programs (Coulsen and Co|;sw/ll 

1965) and especially' in the management of such Instructional programs^ 
(Krlewill, 1969). 



The Establishment of Criterion Levels - 
Performance levels have been used in education for a long tlme^ but 

their use in connection with criterion-referenced tests was mainly 

fostered by the increasing popularity of programmed instruction and 

Mager*8 (1962) work on instructional objectives. 

Since Bloom (1968) popularized the term "mastery learning »" it 

generally has been agreed that the performance requirements should 



represent '^maj^tery'* of the objective being learned* However, the 
particular criterion levels used in any one program are established ^ at 
best, on the basis of experienced Judgment and intuition. 

Although a nximber of writers suggest that they are designing 
instructional programs around the idea of mastery learning, there is 
great variation in the way they have operationalized mastery « A recent 
survey of a number of instructional programs indicates a tendency^tc 
specify a rigid criterion selection policy within a given prograizi« It 
also was reportedly not difficult to find programs "where higher 
criteria are selected in the mistaken belief that this will result In 
.a better quality of. learning product than will a system having a lower ^ 
criterion" (Kriewall, 1969, p. 52), 

The lack of any specific agreement as to the operational iiieanlng of 
mastery learning also can be seen in the way criterion levels have been 
established by those who purport ttT be developing "mastery learning" , in 
their Instructional programs. Mager suggests that one way to determine 
"how excellent (a student) must^ be before we will consider }^m 
satisfactory is to look over the examinations you use« They will 
tell you what you are using as standards of performance ,«." (Mager, 
1962, ppv-^1-52)/ Bloom [(1968) reports haying us^ a procedure siimllar 
to that outlined by Mager. He established his critLrion points for one ' 
class on the basis of the standards used in grading the students in the 
previous year* s classes* ^ 

As Block (1971) points out, at present there are no established 
rules for setting mastery standards and, until there are, instructional 
designers- must rely on procedures similar to that used by Bloom or set 



^he standards on a purely subjeetiva basis « ^Howevery the u$q of each of 
these procedures har yielded quite varied results. In a program designed 
to teach reading skills » mastery has been defined as 95% proficiency 
(Hacfcett, 1971). In contrast, the Individually Prescribed Instruction 
^IPI) project has generally operationaliz§d mastery as 85% proficiency . 
(Bolvi«, Lindvallp, and Scanlon, 1967; Glaser, 1968). The^ Individualized . 
Mathematics System (IMS) has defined mastery in terms of various! 
performance levels dependiiig on the nund^er of test it^ems^and the impor-^ 
tance i>f the objectives to future learning (Ironside, 1971) Although, 

the ^ IMS test authors were asked tc establish ^^^stery scores above t&e 

' * ' ' ■ ' ' ' ■ ■ ■ - ■ . 

75% level, in practice IMS has a range of proficijency levels from 66% 

■ ■ ' ^ ■ . \ 

to 100% vith tlie mo^rt typidal, level be 



Thus far criterion standards have 
arbitrary levels (Bloom, 1968; Kriewal 



ng about 80%, 
been variously set at admittedly 
, 1969 J Merrill, 1971), There ; 



has been no empirically verified proki^dure for establishing the criteria* 
upon which to base the type of instruct lonal management decisions considered 
In this study^ 1 

Tfie Effectiveness o^Mastery Learning Strategies 

Most of the research conducted in the area of mastery learning 

■ ' ' ■ , - " / 

has dealt with the general overall effectiveness of the strategy, ^nch 

research has been concerned with testing the hypothesis that It is 

advantageous to have student^ demonstrate the attainment of Intuitively 

established minimal proficiency levels. There has been no research, ' 

with the possible exception of Block's (1970), designed to deterrainj^ 

optimal proficiency levels or the method of arriving at them. 



Mastery learning strategies generally have been found to be successful 
in terms of both cognitive attd affective results (Postlewait, Novak 

* '■ • f ■ - ; ■■ • 

and Mujrriry, 1964; Biehli^r, 1.970) especially when success is defined in 
terms of the percentage"of students attaining previous grading standards 
and receiving top grades (Mayo, Hunt and Trenssel, 1968; Kersh, 1970)* 
Even though there is no empirical basis for lAatever particular perfor* 
mance standards are used ^ there is evidence that greater lea:mlng is 

• r 

achieved by students required to att1K.n some criterion performance than 

by students for whom no requirements are made (Block, 1970; Lawler, 1971). 

• • ' i 

Heed for Empirical Evidence in DefixJing Criterion Levels 



^ The fact that criterion levels have been established more on the 

. • • ^. ' ' .. \ - ,. ■ ■ • ■ 

basis of the intuition of the individual iristructional designers than & 

on the basis of empirical evidence does not mean that the instructional 

designers are insensi|i:j^e to the need ^or such evidence. On tile contrary, 

many of the writers as well as other concerned educators and psychologists 

have discussed the basis upon which crltl^ribn levels should be |est&bli6bed« 

^laser (1963) has suggested that we nejed to specify the niinimuip proficiency 

levels the student needs before going on to the tiext instructional unit 

in a sequence. Although the difficulties involved prevented them from 

doing so. Bloom, Hastings and Madaus (1971) tiote the desirablljity of 

using carefully worked-^out performance standards/ Hiey also di^uss the 

important relationship between "approptiate mastery levels" and student 

motivation. ' \ ^ 

One discussion which simmiarizes part of theyneed for empirical 

^v^ence related to performance criteria is presented in a manual for: 



10 



jtitute. 



She detemination oi apecific laafttery lavals for various 

subject laatter is aa exparlmntal problem needs to 

be studied. How much mastery j^hould bWrtfuired^^ for 

example j in learning basic arlt|»metic £a^ts 

on to an adhranced toj^lc? Is more rapid learning ai|d 

bett«| retention achieved 1^ 

on in a subject matter where advi^ 

previous lesstnas or is it bestu to require aa earl^^^^ 

level of^stery? (In teacljini typltig^^^^^i^ It 

maty be b«^ to permit the beginning tyT^tst to^^^^t^ 

without cG^itofiQlsing her speed so that ^^ i^^ 

speed and ac^curacy are leartied efficiently .) (^lvin» 

Lindvallian^ Scanlotii» 19^7t p« ^ 

Mastery learning requirements « though only it)^^ 
generally have\been found to, be ^effective. It seems reasonable « then, 
that ah empirical investigation of the most efficient oparatlcmal 
definition of mastery for par'ticuld^ ix^tcucticmai pr^rams could be 
expected to increase further the effectiveness lucid/^fficiency of such 
prpgrsons. , • 

Idealizing the need for some evidence to help esti^llah the 
criterion scores used in mastery testing situatio^a. Block (1970) 
investigated the effects, both Cognitive and affective, o£ requiring 
the attainment of various pre-^established '*mastery*^' levels* . In that 
study, ninety-one eighth grade mathmnatics students were presented 
with a one week instructional program on matrix algebra. The in^roctio 



was^presented in three sequential units. Students wer«^^ rani$omly 

■ .: ' y ■ ■ . . 'V ' ' ■ 

assigned to different grmj^ps \irhich were required to attain and Jaaljatain 

selected mastery levels (no requirement, 65%, 75%, ^5%^ and 95%) on each 

unit as they advanced through the program. Block found that the perfor^ 

mance of es/ch of the mastery groups was greater than that of the group 

0» ... 
for which no criterion level was required. He also found that while 



the' highest, mastery level produced the greatest cogaitl'^^l^ 

It had a negative (^ffect on student/ Isiterests md ftl^ttides; suggesting 

t^t a soicewhat lower crieerion:l*i^isr 

ternw of the balance between cognitive. W : • 

the Block, investigation iiifwB £tom tiie present study In a few . 
ioiK^ttant Vayiii.^ First, the Blodt study was 0 v 

:. ■ . . ,. . ■ . ft . •■■ ■" . >• .»•■ ■ .\- > 

of a relatively brief mA setf-contalned i Instruct ;i6nal progrwi^^^ ' S . 

the gtudet|ts were not: only required to attain a^ glvieqa df gir^ie >of pr^^^^^ 

on each unit , but also to mitintaln that a aipe degree of parotic tency as 

they' advanfccd from one xmit to the next^ Tills is no.l; tWl^ 

paradigm utilised in most instructic^aal progr«»s- ' Itostt Important, - tji^ v 

Bloi:k Study viewed the criterion levels as W^atmen^vy^la^lM 

ths^ decision rules. Thus, there arp corresponding v^thodoXcfglcal > 

differences^ ' ■ ^ .^^ ' . V"^-^. 

The Process of Validating Instru^ional D^ctalbna r. 

To oubclMze tih« afficlency of inatr^ ; * * 

•ona cn^^ence l3 needfid ai|^ 1 
objactlye bef^^re a ^taidaiit^ aucieaafully:^ > ; 

through the Inatructibjaal proigtm mt^ ha». l?^ 

■ itfarnad . BaalcaUy tfi^a i»imiia ^ t^^ al|^Ud!ra»at of aiaa^ttey 
fihlch wlll b« uaad as decision 

In tislng mastery testa : Instructional <iiM«neta/ahould he conc4«(e« -iwjt ; . ^ 
only vith the content validity of ^ thia twta, bwt ^itfih tha> validity 
of the decision rules sftpresented by the criteri<?n scorea. 



Cronbach (1970) suggests that the "validation of a decision rule 
logically requires, an experiment in which after beixjg "tested, person^ * 
are allocated to treatments without regard to the scores ^hose usefulness 
is being validated. The outcootea^ of^^e treatment are then, appraised." 
^Cronbach also suggests that the emphasis not be on a validity coefficient 
but rather on the relet j%nBhlp 1>etT an the outcome measure (s) and the 

4 

> test score. This procedure Is^alid only insofar as the validating 

• »> 

' * .• 

criterion is truly representative of the outcome we wish to measure. 

. jMETHODOLOGY . - 

The specific question^ asked in the present study were: ) 

(1) How' valid, aa instruct ionj^I management decision rubles, jsrei. 
the pi^eset performajice criteria associated with: a selected group of 
instructional objectives (a) in terms of subi^^quent progress through ^ 
ah Individualized instructional' continuum, and (b) in. terms of delayed 
retest scored? 

(2) On the basis of the data gathered in this study, what is the 
apparent optimal performance criterion fot* use with each of the selected 

obj^tly/e,8? ' . "/ ■ . ^ 

(3) jOn the basts of this^study what, if any, aire the^ptobablis 
cost/benefits which qould be expected from the use of criterion levels 
derived by empirical procedures? 

Curriculum Context s . 

To obtain data ^Ich reflect the effects of Independent variable, 
' performance criteria, a nOmber of requlrexn^nts were Imposed upon the 

. ■ • / ■ ' ' 



i^rlculum context In vhich this study was conducted. It was critical 
that the sttidy be conducted within the context of. an instructional 
program which makes instructional management, decisions on the basis of 
'^mastery** scores, /The'^^rogram also should have been field tested to 
Insure that it was operational ai!kd generally effective. 

The program selected for this study was the Individualised 
Mathematics System (I^). This pti^ogram^wa^ ^evelo^ by the National 
, Laboratory for Higher Education and field tested during the 1969«*70 ^d 

1970-71 school; years (iFtary, 1971) . IMS' uses a variety of instructional 

/ * . , * . *> • 

techniques to teach the many objectives found in its purported hlilrarch* 

0 , ••• 

• I- ' - ' ' . • • . 

ically and logically sequenced continuum. As in most .individualised 

\ ^ V 
programs » student progress 1? controlled by the decisions made tm the 



hAsts of the posttests administered at the .end of each instructional unit< 

In the IMS program each unit is composed of a number of objectives. 
Although the posttests for all ilnit objectives are found at "the end 
of that unity the advancement-recycle decision la ma^e on the ba«ils of 
each separate objective 7>osttests. Thus« after taking a unit, posttes^ , 
a student Is recycled through that part of the instructional program 
related tc those objectives » and only those objectives s f or whlch^ he 
failed to achieve a "mastery** score/ 

Operational DefinitiottS ' , r 

For the ipurposes of this study the students* ppstinstructipnal ^ 
proficiency Ifevels were measured, by the* posttests suppU^d by the IMS 
li^eCructional program. Reter^tion was measured by the u^e of parallel 
tests developed using the statement of the objective and the existing 
tests of that objective as guides. An indicat^ion of a sttfdent's- ^ 
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subseq^ue&t progress through Ihe program was obtained from his posttest 
score Ion each objective in the next sequential unit of instruction. 

J ■ * ■• ^ - . .. .^L... — 

Setting ; 

This istudy was conducted during the last half of the 1971-72 school 
year in the ABerlcan Elementary School lil Karlsruhe, Germany, as part 
of that school ^8 overall attempt to individualize its instructional 
program. The Karlsruhe school had ^ enrollment of about 1,000 pupils* 
The children came from a variety of ethnic and economic backgrounds. 
Most were the dependents of Army personnel, both military axu3 civilian. 
Being the only American school in Karlsruhe , it elso served as t^tk^ 
educational institution for the children of most of the American bus inessmen- 
residing in the area. 

Subjects 

The subjects used in this study were selected frdp^^^ 4^ those 
pupils, at Karlsruhe who were working on a specl^ed. subset of IMS 
objectives during the time the study was being conducted. it siMm 
reasoiMble to suspect that tiiiB sample of i^lls working idf these units 
during this given time period were representative of the Kkrlsruhe 
pupils for whom the units were appropriate ^ A fur^heir description of 
the selection of the subjects is presented in the next section. \ * 



Research Design and Data Gathering Procedures ^ 
y This study was designed to. Investiga^te the validity or the go /no-go 
decision rules associated with the posttests found atythe end of a ; 
selected set of inst;ructlonal, units in a sequential individuaIl>a(^progt^ 



/ 
/ 



The general proc^edure 8 followed in this 8tudy>vere those suggested by 
Cronbach (1970) £or the validation of decision rules* To study the 
accuracy of the decision rules associated with' a pairt:lc\4ar unit* t 
^subjects who took the posttest for that unit were advanced to the next 
aequentjjLal unit without regard to their posttest "scores on the fixnt 
unit. For convenience these units are designated liere as A and B, 
respectively 9 with subscripts indicating^the different pairs involved 
in the study » A retest Of ,the unit A objectives was administered as 
soon after the unit B posttest as possible. The data gathering design 
for each pair <of units followed this. i order; 

Unit A Unit B Unit A 




Instruct ion-^Post test • Instruct ion-^Post test Retest 

Figure 2 shows the units within the IMS cont^tiu^im wh were 
selected for stu^. As shown in Figure 2/ the IMS continuum consists 
of ninety units organized by topics and levels of difficulty. Each 
unit contains from one to eleven objectives. Generally, a pupil tooves 
through this sequence in order, that is,c from top to bottom and left 

to right, each unit purportedly building upon the preceding units. 

■ , . ... * * * ■ - , , 

, The four program segments shown in Figure 2 were selected on the 

basis of the identification of the concentration of dependable student 
data^ the availability of instructional materials and. a growing awareness 

of the difficulties involved in the collection of dependable data. 

Implementation of the described research procedures was initiated 

on January 18 > 1972, with the actual collection of data beginning on 
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February 7^ 1972, Data collection was complete.d on April I7th of that 
year. The data consisted of posttest scores for each objective In the 

first unit (A) of each pair, posttest scores for each objectivelin the 

■ »• 

second unit (B), and retest scores on the objectives in the A units 

I 

obtained from carefully constructed alternate forms of the respective 

posttests* Table 1 shows \the number of objectives Included In each 
of the units studied. ^ 



Ten classrooms wer^lnvplved in this study. They consisted of all 
of the 4thy Sth, and 6th graie classes at Karlsruhe except for those 
few in which the IMS program had not been sufficiently implemented to 
provide a truly representative IMS classroom situation. 

Data were collected on all pupils who were: (1) in one of the 

h 

classrooms where data was being collected, (2) working cn the specified 
units cluring the time the study was being conducted, and (3) who, upon 
completing a give^ unit A, were to advance to the next sequential unit 
In the program. 

The teachers and paraprpf essionals at Karlsruhe were dlr^tly 
involved in implementing th^ design procedures and especially the data 

collection activities required in this study. The teachers were asked 

■ . . fi- ■ 

to make certain after a subject completed a Unit A posttest he did not 
do any further work in any IMS material for that unit« However, the 

aubjects were^ exposed to the ongoing instructional i^rograms presented 

■ ' • , . ...... . ..\ - ■ . ' 

in their respective classrooms. At times this includied exposure to 

material on topics related to those which they had ju'pt studied. , 

After completing a Unit B posttest , each subject was given the 

appropriate retest. The teachers and paraprof essionals were^asked 
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to adi^inister the retest as ^oon after the Unit B posttest as 
possible. Often this was either the same day or the next. Due to the 
record keeping procedures In a few classrooms It was sometimes two or 
three days before the retest was taken by a subject* Due to the 
continuous monitoring of classroom activities » testing procedures, and 
IMS records, It was ^ever longer than one week between Unit B posttests 
and the associated retest s* 



ANALYTICAL PROCEDURES AND RESULTS 



Validation of Program Performance Requirements and Identification of 
Optimal Criterion Levels " 

The decision rules investigated in this study were those associated 

with each objective found in the selected four IMS units^ Fractions IV, 

Tiae' IV, Numeration V, and Multiplication V, Data were gathered on .a- 

different group 'of pupils for each of these four units<» Table 2 shows 

the number of subjects for whom data were obtained for eacdi unit. The 

^ " ■ ' ... 

four' selected units contained a total of twenty objectives. Thus^ there 

were twenty decision rulcfs whpse validity were investigated in the 

present study. For convenience » these objectives are often referred to 

by their respective reference numbers » as presented in Table 2. 

In analyzing the validity of these decision rules the scores on each 

objective ppsttest were dichotomized at all possible decision points. 
For example^ where an eight '*item posttest was involved, the subjects 

were classified as go or no*-go on the basis of those who scored 8, 7 or 

hl^gher, 6 or higher and so forth. Each of these dichotomized groups^ 

which represented different decision rules, were cross tabulated with 



Table 2 



REFERENCE NUMBERS, CORRESPONDING IMS DESIGNATION, NUMBER 
OP TEST ITEMS FOR EACH OBJECTIVE, AND NUI4BER OF SUBJECTS 

PER OBJECTIVE / 



Assigned 

Reference 

Number 



IMiS Designation 

Objective 
Unit Nxambdr 



Number of 



Asi 
Te! 



ociated 
t Items 



Numbef of Sub j , 
for Whom Data 
Were Collected 



A~l 
A-2 
A-3 
A-4 
A-5 

A-6 
A-7 
A-8 
A-9 

A-io 

A-11 
A-12 
A-13 
A-14 

A-15c 
A-16 

h-n 

A-18 
A-19 
A-20 



Fractions 
Level IV 



Time 
Level IV 



Nvuneration 
Level V 



Multiplication 
Level V 





8 




2 


9 


■■ " N « 62 


^ i : ■ 


8 






11 




5 ■ 1 - 


7 . ■. 






.3 > 




2 l 


6 




i 

3 1 . 


4 ^. 


K " 49 




4 






6 


. ■ * 




6 


. "f ■ . 




. 4 . 


M « 31 


■:/■- 


. - 7 " - \ 






13 
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6 
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N « 26 
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8 
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each of the dichotomized validating criteria to form a series of 2 X 2 
contingency tables and a contingency coefficient was computed for each. 
The^e contingency coefficients were then used to Judge the validity of 
the IMS program decision rules and to Identify the apparently dptimal 
decision point or mastery level associated with each selected objective • 

For clarity, the meaning of the telnns performance standards, ^.piriterion 
level and mastery level, which are used interchangeably, need to he . 
explained in the context of the present study. Most of the discussions 
about mastery levels define them in tl||^^f some percentage score. In 
many instances these definitions or descriptions of the required performance 
levels are misleading, at best, when translated l^ito actual program 
r operating procedures. For example, suppose an instructional program ^ 
purports to be requiring a 95% criterion level, but is using tests of 
less than twenty Items. Then that program^ in actual practice, Is either, 
requiring a criterion level of 100% or some performance which might be 
substantially below 95%. \ . - 

To avoid dlscrenimcies and because of the variation in ^he length ~ 

of m^ny IMS posttests, performance 8t:^dards are reported for this 
study in terms of the number of torrect Item responses required. Thus, 
ifhen the program performance criterion for a given objective is repotted ; 
. as belng'^^ve^^l^ Ite^s ^ 

correct oil that post««^test before he wo;}ld be advaiM 
Similarly, /a suggested optimal criterion level of 6 on that test means 
thafby accpirdlng to the data obtained in the present investigation. It ^ , 
apparfi^Hy would be maximally efficient to base the advancement decisions ; 



ERIC 



22 



associated with that cbjectiye on whether or not a score of 6 or higher 
had been attained. To inake these performance levels meaningful and 
interpret able, the ninaber of items associated with each objective is 
given in Table 2. 

kB represented in Figure 3, two criteria were used in selecting 
the optimal decision point associated with each, objective, (a) there 

/ must have been a significant <cr< .10) contingency coefficient , 
between scores dichotimlsed at th^t point and at least one of the 
validating criteria; and (b) the/number and magnitude of significant 
contingency coefficients between it and all validating criteria must 

-"have been maximal, i.e./ greater than any set of significant cotA:lngency 
coefficients associated with any o^her decision point. * 

If, and only If, the IMS' decision jpoint was optimal ais defined 

. above, was it designated valid as a decision rule . l£ the analyses for 
a particular objective produced an optimal decision point for that 
objective which was other than the one given in the IMS program, the ^ 
program decision rule was designated invalid, and the optimal decision 
point was designated as the valid decision rule. <st 
If the analyses for any given objective yielded two or more 



decision points the number and magnitude of wl^se respective significant 
contingency coefficients were approximately ^qual, and if the IMS 
decision point was among themy it wais designated ylalid as a decision 

rule. If, however, the IMS decision "^polnt/ was not among the cdnt ending 

'•'■■•-.■•■'/., . " .,' . ■ ■ 
points, that contending point which differed least from the <IMS decision 
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point was designated the optiiaai and valid dec Is ioxr rule, this procedure 

ii|inimlzes the number of students affected by a change in mast^ary levels, 

■ ■ ' • if . ■ 

ifhen that change is between cotspeting decision points« It t'i^i;^^ yields 
the more conservative estimate of the cost /benefits discussed in section 
three of this chapter^ This procedure al^so lends somo weight to the 

intuitive kioowledge of the instructional designers as reflected in the 

■ ' .. , - . , ' r - • • . ■ 

a jgrlori program toastery level. 

XI the^litialyses for a given objective produced no statistically^ 

* ■ ' . . ' ■ • ■ ■ ' -' - ' ' 

. aignif leant (a< .10) contingency coefficient h.e^ ween any of its . 

possible decision points and any of its' validating criteria j jthe optimal 

-decision point was considered indeterminable.. 

To clarify and illustrate the analysis and reasoning used to 
Investigate the validity of^fh^ selected IMS declsion rule^^^^ 
identify the apparently optimal criterion level for eac^ objective; a 
description of this process is presented for two objectives/ These are 
the objectives f6und in the WS unit Fractioitis Level IV^ and designated 
ifs objectives A-2 and on Table 2. 

in^ables 4 and 5 only thoise decision points are presented for 
which ttfiere wefe cocnputable chi squares, since without them no contingency 
coefficients could be obtained. A chi square would not be ^omnn^sbXe > 
when any two adjacent cells of a 2 X 2 contingency table are empty, A 
non^computable chi square can be interpreted as a lack of any predictive' 
power bf&tween the posttest scores dichotomijied at that level and the 
associated validating crlteridn. As/ shown in Table 3 » when there where 
no pupils in either a column or a row there is no relationdhlp, between 
the decisions made on the basis of the posttest scores and the dichoto*- 
miMd scores used to validate those decisions « 
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Score on 



Score on 
Boat-Test 



1 « ttttifiin^^ 

0 « failed to attain ^masteiy*^ score (re-cycle) 



vhllfi tlbttire if « «l|ppttif leant 'r^^ beitir«eii debi^ic^^ 
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Table 4~_ 7) 



C»NTINGENCY COEFFICIENTS; CONTINGENT RELATIONSHIP 
BETWEEN THE ATTAINMENT OF SEVERAL SUBSEQUENT 
PROGRA24 CRITERIA AND SEVERiii:. LEVELS OF 
PERFORMANCE ON OBJECTIVE A-2 



Objective A-2 ^ Subgeguent Unifc Objective 
^ Level Re-Test B-1 B^2 B-4 

, g ■ ' ' ' ■ ' ■ ■ ' ■ . , I II I . — 



9 , 
(8) 


.2878*^^ 


.1641 
.1495 


.1173 
. .1483 


.0197 
,1028 \ 


.1289 
.1385 


7# 


.3672** 


.2860** 


'>25^4** 


.1244 


.0999 


6 


.2740** 


.2449*^ 


.2384* 


.0822 


*04S7 


5 


.2653** 


.2359* 


.0910 


.1415 


.0804 


4 


.2591** 


.1327 ■ 


.Qf5j6^8 


.2263* 


.0171 


3 


.1914 


.1736 


.1450 


.240^** 


.0397 


2 


.2534** 


.1342 


.0187 


.0139 . 


.0847 



^ ^ 9 -J — : — . : _ — — . 

N « 62 ' • * • "" 

* p < .10 * 

** p < .05 • ' . 

I beside level indicates apparent optimum 

() ^ parentheses designate required program mastery level 

«clv«nc«d to the next unit, and then performed satisfactorily on the 
flrat 0«t of posttests for that unit and attained the required score 
on the'^retest. A recycle decision was also labeled as "correct"' if» 
hid that pupil been advanced,! he would have ^'failed*' eitfier one of the ' 
posttests in the next sequential unit of instruction or the retest. 
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Following the previously described guidelines » level 5 v^^s identified 

as the apparent optimal criterion level for Objective A-5 (see Table 5). 
Of the contingency coefficients associated with Objective A«>5/ those 
computed using critefion level 5 are the largest and are significant for 
four^^flfths of the validating criteria. 

Utilizing these sanse procedures^ the validity of the program mastery 
levels were Judged. Apparent optimal criterion ,levels also were 
identified for each of the selected IMS objectives. Table 6 presentts a 
summary of the results for all twenty of the inyestijgi&ted instructional 
management decision rules. 

The results summariz<sd in Table 6 offer evidence to support the 
validity of three of the twenty selected IMS instructional management' 

• . ■ } 

decision .rules. The data indicate that, for these three objectives^ 
the program established criterion level would maximize the number of 
correct instructional management decisions. No change is suggested for 
die criterionV level associated with these objectives. This is indicated 
by the word "none" in the isppropriate column of Table 6. 

' For f^ve of the selected objectives nc» evidence waa obtained in 
the present investigation which would atfggest tjhat any of the possible 
decision points 'provided faly the respective IMS tests could be of value 
in predicting suhisequent jpupil performance^ - 

As shown in Tahle 6^ the data indicated that a change^ in the 
criterion level required for the remaining twelve objectives would 
optimize the relationship between the decisions made on the basis of 
those criterion levels and the respective validating criteria. Of those 
twelve suggested changes, four require an increase in criterion 
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Table 



CONTINGENCY COEFFICIENTS : CONTINGENT RELATIONSHIP 
BETWEEN TKE ATTAINMENT OP SEVERAL SOBSEQUENT 
PROGRAM CRITERIA AND SEVERAL LfeVELS^ OF t 
. PERFORMANCE ON- OBJECTIVE A~5 



Level 


Re-Test 


B-1 


B~2 


, ,B-3;;:..;..^^. 


B-4 


7 


.1140 


Vi781 


.0324 


' .0564 


.I352 


(6) 


.2634**^ 


J .2236* 


.1764 


1244 '''M. 


rl631 


5# 


.3546** 


.3279** 


'.2162* 


■t.l733..^^::/ (. 


^212:4* 


.4 


.1134 


.0750 


.0153 


.0766 . 


► 0219 


3 


.1911 


.0283 


.0261 


.0342 . 


.0730^1 



N ■» 62 

* i P < .10 . / 

** p < .05 

f beside level indicates apparent optimum ^ / 

I) parentheses designate required program mastery level 
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\ 



ratio of the nusiber 



8 who would have 



performance while the other eight indicate a lowering of the required 
performance levels* 

Table 6 ahowa the number and percentage of pupl 
been affected had the suggested criterion levels beeix used rather than 
the prpgr/ua established criterion level. Th.e figure: i in the percentage 
column of Table 6 represent, for each objective, the 



total lik^j^ aubJiili^Mi^ tb thvtt objective wer^i 

collected. As shown In Table 2, this last nuaibet va::ie8 depending oA thct 
objective. These figures provide an Indication of tjhe degree of <^ InstnictlOMl 
efficiency to be gained by using the suggested opt Imiil crltex^m {K>lnts. 
These data can be used to estimate the cost/benefit ^f empirically derived 
criterion levels. 

Cost /Benefits 

In Investigating the valu^ of empirical proceduraa for deriving 
criterion levels, it Is necessary to examine the cost /benefits which 
could be expected ^rom stich procedures, this entallii an estimate of 
the Instructional efficlencyf which would be gainjsd fi{:om using empirically 
deriv^ criterion levels. Tb accomplish this some reasonable approximation 
Is needed of: (a) the cost of each Incorrect instruc;tional decision, 
(b) the expected percentage of increase in the Gorre<^t instructional ; 
decisions using empirically based criterion levels, aind (c) the number 
of instructional management decisions made per year ,|^r child in jthe 
operation ^f the program. ^ i 

The cost of each incorrect instructional decision can be estim^^ted 



by multiplying the cost of each instructional period 



by the number of 
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instructional deeislocis is oooiputttd heti on the bMls::«^^ 



'iff. 



to the operation 'Of the Karlsruhe ik^ 
best indication of^ the per pupil ebit of e<fticatJLon to 

one...3rear...is:.-;reflect^d- in-the'V^i^^ 'iu|J^;;^^ " 

is charged by the Department o£ the ^Aray ti^ 
wishing' to' enroll ^'theiV d^ependents/: .As wl 

there are' 'approxiioately ''isix instructional - periods ' per '^'d jir^,'. fSenK^^ '~ ' ' 

the 180 days.of school, o^ a total ti^ 'i^^ 
on observations and anecdotal "cofi^nts there usually di^ 
day period between the tine a pupil ia recycled and th*^ tj^ftit^ 
that post-test again. Using this information the cosr 
ilkstjructional decision is estimated as: 



■■■■ vi?!: 



$875 

Cost /Child/ 
Year 



$ .81 

Cost/Child/ 

Instructional 

Feriod 



1080 

"Ho, of 
Instructional 
Periods/Year 



Instructional 
Periods involved 
in Ea^h decision 



■1 



The results reported in section one of thih eha|rt^^^ 
for twelve of the selectfld objectiyes the suggt^jtjpi 
criterion level Which is differ«mt f 




Tiibl,« 6 •hows tha nttad>«r of subjiiet* «^ 
•the guggested. opt inutl criterion -Xn^nX^^^iti^ AatilXM^^ 
progru. established nastery levels. ; A'.; totaling of-.. thtfi'-o^ltte 




also shows that altogether 91 advanca/recycle Instructional 
dadaions would have been affected. This represents a little ^re than 
lOX of the total noaibar of Instructional decisions conslderad ^^^^^^^ 
study. Therafora, on the baal* of tli« data gath^ad li^^^^^^ 
ft !• astinuitGd that, uaing empiric al^ia^^^ itaada^^i;^ 
tha accuracy of approxiMat«ly of ttii^^truct^^ 



Aeeording to Frary 
conplata about tbirt4«ai 
appmximataly f tve objectii^am^:::^^ 





a^waga, pupil wlU-eacoimtar,,alirty^;:jd|^ 



"'dorii!^,. a ,glven;'achool...,year., ::lf .tlM|i|^,i-=9f 
lavala Wputd Iqc tease the accuracy by '^(^^^ 
- vottld//ia*4m'':tha 
tine par child per year. 

This dollar value is baaed upon an ii^st^t:^a| 
which in, turn, was derived from''an- Wttal..p#rr.i^^ 
' 108Q instructional periods, • three of <iiiich' wa^^ai^^ 
v'ioat'ructional ^decision. Variations -in ' attnual^:^r Jp«^il 'ia^ailuraf! • • ' •,, 
would directly affect the astliMited cost par chlldrf^bttt vatlft * 
tha lengtl^ or nuod>er of instructional period* prpbalily would irot^ sii^^^ 



euK bij^tlyeji^^^ been used to deci de v hether or ^.il vas ready 

to advance through the Instructional sequence. Thus^ the mastery levels 
associated with e&ch objective functioned as decision rules used to 
predict subsequent pupil performance* 

There were two pritaary purposes of thev present: study Cta^Vf A to 



1 . " . .^fv 



iirvMtlgate the v^^^^^ 

IMS instructional objectives. The other purpose was to*^td<pf^gitItyt on the . .v^: 



basis of the empirical data gathered, that "criterion lev^l lor each ' ^ % 
objective which could be expected to optimise the nuiaber of corri^i^t - 
\^ instructional decisions. ^ j 

The procedures used in this investlgetlon appeared to, be fairly 
successful in identifying the apparently optimal critetionlevel*^^ 
the selected objectives. However, only thrWe, or 15%, of the opU 
criterion levels Identified by the data were the $aM as thiM^ 
levels intuitively established by the progtto* 
used f>y t|i« instructional deaigneri? were nic>e ^i^r «cei^«t#"'^^^ 



optimal criterion levels. '°''Ap|Mirently^ .e^'''U«eoi>i^:;^pUtea^^ 



make a substantial improvement in the process of elt«blishing ct^^^ 




levels. 



Kaal^ of the current ly~ available ixiMvU^aXU^-^ixtisbt^^ 
requlrethat sasM criterion or mastery level, of ten J«)f ined. «s . 80S^ pi»»££^l«^ 



^ItfiiMt u^^ Tba results of thl« tovMti|«cii¥ti:.did. nilit^i . 



eUf^port to that praeticea 



In this & the suggested optitaal cl^^ter^Oja JLe^ 



|p.:'i ■ • in- Aw^^^^^^^ wa'the basis of ^stlgiiifieMiic. relaticNMl^ with --^ir/iy^ 
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4mly-oi»a-or two of- th« «aiidating ctltMia>-i;Jlkea^^ 

(KsplAnations why »ignif leant r«lAtion8h*p» w«r« not pbUin«d iBoro/^^^^^ . 

I' 

itw^vakXy. first, the d»««nc« of any stgnlftcant rolatlonahlp t^^twaan 
a given ol^iactlve poflttest and certain of thi respective valldaUng 



crtterla mlght simply: be reflecting the A»iJLoijm-%sm^t^'^*od«i^'--,^^;-^^X~ 
oi relationship between them.„ ?or example, there v^^t ^|«tiOttship 'iS^i 




a'stobsft^iiim prollctWy on'one of the specif l«r^>U^c<:lti ^^^^ 
It is not, however, reasonable to suspect Jthls as, «^» «^» MM*e ,y _ *f 
Often low 4«ul^««etl«es non-slgnlfica^ relatlorij^l^^ySlrfC^^^^ 



^•ttest-«nd the retesrdf t«ae, objective^. 8»w1i'iw4^l^^^ 



the T«test'':-ttt^-ti»«c;f5003ectxve^«^^^ ' 
e^ingencl^s mre explained V «h;ft ise^et two factors 
redisce the obtalneid contlnigfliv)^ 



the s^cmid factor tihWxedtjiced th^'txtixO^i^^^ 



coefficients hetween pttsttest scores and jaIi^JS| 
f rem the fact that this study wa^ c<m^iiet4Ml^ 
operating conditions. Althc^ teachers 
after th* subjects completed msf thiit A i^ostt«»stC^tJW W fWf^^^ 
further work in any IMS material for that uittt,. tl^j»tthjecfi>ersuesfflosia 
to tta« ongoing inatructloual pro 



teachers reported ^ai:« ^.b»oa^ it iiad ' ~^ ^1 



classrooms. In addition, a few teachers reported ^at//b#eai|* It iiad 
becoite part of the^ regular operating grocedures, they^^ OT^^^^^ 
might have gon^k^ over missed test litems with some stibjeets before 



advancing them to the next unit. Zb» ocettrrenee of ttkUra^lti^t^A 
inatroetion could be viewed as a tasthodoldglcal weakness lAl^ light/ .*:i< 4 

have been reduced bad the s^udy been cffadUcted uhder mote JiAfiWo*^ r ^ ^.^j ? j 



Voriene«d candltlonsv H *he presentation df thltf Instruct ion ii; 

typical of the coi^itions that could t>e expected from any real cXaesrooia. 
aitiMtitiai ifhere t^^ been iii^l<pent^d. Thuov ^ ^laa^^l^^^^ 

ar$ued that the presence ^^ 0^ 
^ not d«ract ^rom,^^^^ 

collected from this inve6tlgat£on» 

Ihe third f actpt the ttoaher of ^^ijpiif leant 

contlngeiuey coefficients obtained was the IMS tests themselves. The 
study vas conducted within "the conatraints imposed by the length quality 
: and thus reliability of the IHS posttests. In relation to this ^ it 
should also be noted that the selection of the optimal criterion level for 
any given objective was restricted to those levels ^e possible by the 
length of the corresponding IMS pbstlest. It is quite possible that^ ii^^ ' 
tests coulii be deslgued which would not only have cWeo&t . validity but 
would eiso have greater reliability isnd thus would be mot^e 
"-iwhatevfr relationships did^^^^e^ between -proficiency da the^W^^ . 
objectives aoad the correspoxidlng v^^ 

One of the main rationales for instruction 4eaignedjf^^itt^ lidea g^y 
J<>r m^^ at least 9W of todey's learnisi^ii are ca^le, 

w of attalniiiig Itie desired goals an instructio)!^ prcfj|raaf^f only they 
given sufflcl^^ 1968), A^ordingly, one^^ 

^uripose of tsastery learning instruction is to vreduc#\ t^^^|^jpM^ ; ; ^ 
of fiilure foiind in conventional in^nructlon. has bewi juggested^^^^^^^ 
the ellittiiatiott of this fail^ especlislly in tht el^aentary .gtikdee;^ ^ 




would have %:substan^ positive effect on the selfrcoticept^^^^'l^ 
que^ e large segment of ouir^idipc^V^^ 
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(Bloom, 1971). Specifically, this Involves those pupils who have experienced 
repeated and continued failure under traditional Instructional practices and 
Sroding procedures. Unfortunately, the data gathered In this inve^jLgiatioa^ 
•uKgest that, although no longer graded on the basis ^f JccKt^^etition with 
Other pupils^ some p«P_tl« still ^xjt$rl«^«& f 

Aituations. In the IMS F^^oglfam^hls failure takes the form of/h«liSg fieeyclea 
ItMt recyelitiff was^intMo^^ ; 
•ttltudinal and motivational effect iras evidenced by %mifeodotal eonanettts 
of^teachers and pupils. These negative alde>e£fects of recycling have ., 
also been f oiu^ in other mastery leainiing ln8truc;;iimal ^itt^ationa (BIqns]^^^^ v ;>v 

1970; Lawler, 1971). .V / 

•■ ■ - • ■ ■■ . ■ ■■ ■ \ .-■ ^ '\ ^ V 

These findings eaiphasixe the itB|K>rtenfce of identlfytng (»0e£o#|U|^^ 



criteria which will mialmi»e any uimeceasary recycling of j^Vi?^* 
segments of the instructional program. In addition. If all tm]^tlV «<t*. '^ 5 

to attain <'mastery«» of the required objectives in a given ijiiflfrttAtiorti^ 

' ' ' . • • ' ' ' ' / \ c^hi ' ^'/V 

program without experiencittg the failure associated with xecycHitttiKf^ 

adaptations need to be made in that program. One possibiUty wpuld be ^ 

to accomosodate individual pupil differences in ability through tbe^^^^^^^^ -^^ 

quality or quantity of instruction provided. The quality of instruction ,4 

however/ is very difficuV: to regulate. The most common current practice i - 

in individualized instruction is to vary the quantity of instruction 

largely by adjusting the amount of tim^- instruction is presented 4 To . 

minimii^e failure experiences these differences in the length of exposure ' 

time need to be made before the pupil takes the posttestt rather than 

on the basis of how maiiy times recycling occurs « Another alternative is 



I: 



^ u -.i-^- ^ , , : — . 

:i tp- esemt Instruction in a manner so that the pupil la ^avrare that 
' ' reeyeltng or remedial Instruction haa.ooeurred. . 

' the eai;iiiiite^ Incre^a^e in Inatruetlosal efficiency ieport#d:.ln the > 
' coat/henefita-a^tion indlcatea. that there ia^a aMhatant^al ititatruetitti^ 
. gaitk €0 1m expected froa the utlliaatlon of eoplrically derivad c^t^tl^n ; 
levttla In^ ncSking instructional laanageaent deciaiona. Iftietheir thia 8^1^ 
la aufficlent: to Justify the coat involved in obtaining theae criter^lon 
levii^las however, depends upon the conditions uider iwhich^h^ e^ 
data are g[athared. For the typical school or diatriibe the required datai 
: gathettjig and procedures w^uld: he far too cpstly; ^^nvolving 

resources not generally available to then^ ' ^ ^ ; I 

jp<>r th^ identif ic^tipn pf optimal cri|:erlo^ l^ela ta be feialM^^ f 
And coft^effl^i^fint ^tfa||ra are^at least thtfe i^4^ar^^<ni^^ "Jj^f^^-'!^^ 
0<nnatralLnta.cf(i be m^ beat b^ the Is^iitsctitmal '^Ni^^ dtn^Lng a 



^0-^^^:^:^ •^;^■:^^^v;r■;:^^^v:--■ .-^.....v^^^^-^^v- ,v:. ■^-?^;-'^';.:^,ii^-:v 



vfomuitlv^ «Valumon fiold teat of the^iwrcigratt. *iTat,,tha firqffeeai %| ''^^^/^l^^* 
idrnti^ltytos optimal criterion J^i^ala, n^e^ife^^ba^tea^te^ ; 
^ kimirledgeablo in educational itieasureittent^lM.%ifati;^afc*^»,^^ SlM^^d^ 
o . typically found in iin^|«aitjte$ atid aduc^i>iM5l 4ab<!tratwl^|^i»r« . .-^"^£"^4^ 
infttructioi|al;,prbgr«s»a jaxie iwLts^ 4ev4^|bs»i|d«^ 




g^t^ering putj^oses during a formative eyaulat ion ffeid tje«fe 1» 

If a cbmpttt^r is ueisf dur^ tbift ji:ime» 'it would 
easy to adjust the programsfllng so that the data needed 
for lieiitifying «jpWttf$l ttrtterion be ^athetpd, tAitv. 



!^^^ faasible and cest efficient Identiil- 




b« anatcized over a large niaaber of use^s. Again, It would >ften^ the 
beat way to accoBpliish this is for the initial cost to be Incurred hy 
the instryctlon^l deeigpaers and then spread ovet all program pwir^ha«*fcc. 

Jxi coi^lLuslou, the methodology aod z^aulta of the preaem^ iiiv^ 
«t*g;^!<^^^^^ exploration in three additional separate, 

?^ areas of x^eilrch ' 

relistia t<> l^ke fiaasibiU the type of investtgation re^tted 

^^er#^^ setting using CMI, In scidh a'ail^iatl6% 

It ptg^ to investigate lAat test lex^ths and itte 

^arfii<^ fum:tlcm tuost efficient ly ^atid yield the great^est predtetlve 

' as liel^ validity. 

^ second area of research involves determining ^et;her the 

ppt^^^ for a given objective is affected by the type of 

: 4e^^l»g <requlred in that -objective, In other words* before a pupil can ^ 

■ - . ■ ■ ■ , . t.- ■ . • ... ^ ju: • 

be; ex successfully advance through a hierarchically structured ^ 

iil^ttticti<mal program, does he need to demonstrate a different degree of 
I^r0ficlency o^^ the attainment of verbal Infornation 

-tihan^^^im the attainaii^nt of verbal inforwation th^ 

4m^^ use of dbme intellectual skill? tte moat 



«ppiiropri^4 eotit«xt for «Dsi»rl^ this qu««€loB ffight h« k •elwsbtt 
«tinrieai«ai, diii to'ths large nuabar of bdth fcypfts of Uaraifl^ r^^ttlrod 

instru&tton^I daslgiMira, nted ft* 
pore eons id«r«tl<m to a ownbttr of laportaitfc Initru^tiMat qooatlons: 
What is ni(«t«r7? Why is aaatftry, as 4«f infttS itn the prosran, ra^cdrti^ . 
for « particular objective? By wh*t standards caa Instrttctlottlll doslgnera 
Judgo i^ethttr the attainaeat of "oastaxy" oa a givan objeetiva aarired 
th« daslrod pwrposa? To a vary real degree, this Itwjlves tdeattfyiag 
criteria which can be used to Judge, at least partially, the' valve ^ 
well as the effect lveae'86 of reiiuirliig nastery on particular ohj^tlves. 



Current discussions about ■•»a«tery" learning and criterlMi r«f^ftneed[ 
Instruction reflect great diversity in the way crliberioii (or *fiiMKst«ry*») ' 
levels have been def lifted and determined. CrltWion levels within « 
jPMgrn are vleMd as decision rules used to Judge lilen a ^j^il .is r^edy 
to e4v«iice to the next step in an Instructiimal seq^oeneW 

ibe pis to investigate the ^ijl^^"ia 

isMtructioaal^nanageaent decision niles, of the preset W«o^ «i;it«rlott 
leivele CSCL) associated with a selected, grolip of inst^tional ohjeetim. 
in an individual^ed natheiBatl«8j;9a^^ 4:? tetCNpie^aditxer foie; 



e«plrlc«liy deteraii£ing optlnal KXs, The vaUdation vks ioptfueted 
tenis of delayed retest scorea and in teroa of .«ub9iNftiii^pi»f««i«; , 
thvoiqih the innruetlonal eontinuun. , subsequent |»ro^s$ was' e^e«Rtiton«lly 
defined as attaining the re<ittired ^perfonu^e level on the ohjecitives 
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found la thfi next sequential unit of inBtructlon. The empirical data 
gathered v^B used to suggest an optimal criterion level for each of the 
selected objectives. 

This investigation was conducted in tfhe conte:xt of the Individualized 
Hothesatics System (IMS) as implemented in the American Elementary School 
in Karlsruhe, "Germany* Subjects vera selected from those pfi^ils working 
OKI the selected subset of IK3 objectives during the time^ of the study. ^ 

Tc investigate the validity of the decision rules « expressed as mastery 
levels aasopiated with the objectives in a particular unit, the subjects^ 
i9ti0 took the posttest for that tmit were advanced to the :susxt s|kqueEntiat\ 
unit witl\out regard to their posttest scores on the firat <mit« A ret«f t| 
of ei^h aeled^ objective was administered as soon aft'er the subjitOta 
completed the next sequential unit as possible. The^ date collected y 
consisted of posftest scores for each of the selected object iveSfdel^ 
retest scores on these objectives » and posttest scores for each obJe^Mve 
in the next sequential instructional unites , 

the decision point selected as optimal for cefch, objective was the 
onejffaich yielded the largest aignif leant contingency coeif^c lent s with 
-the greatest ntinber of validating criteria « The validity of the a priori 
mastery level ei^tab 11 shed for each objective was Judg^jon the basis of 
whether it was the one selected as optimal for that objective. 

The procedures used in this investigation appeared tb be suceessful 
in identifying optimal criterion levels for the ielected objectives. In 
terms of validating the program mastery levels « however j only 15% of the 
optimal criterion levels idc^itlfied by the daCa were the same as those 
criterion levels originally established by IMS. Thua^ the procedure^ use^^^^ 



hj the Inatnilctloiial designers vere not very accurate in* identifying 
optimal criterion levels. Apparently-^ the i;ise o£ empirical data can 
Mice a substantial improvement in the process of establishing, en- route 
criterion levels. 

The increase in instructional efficiency estimated in the cost/ 
benefits section of this report indicates that there is^a substantial 
Instructional gain to be expected from the utilisation of empiricMly 
derived optimal criterion levels. For this gain to Jusify the cost 
involved in obtaining these criterion levels, however, it Is suggested 
that the data be collected by the instructional designers during the 
formative evaluation field test of the program^ using automated data 
processing and multlple-matrlx sampling techniques.'^ 
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